Papers with anisotropic embedding space
An Isotropy Analysis in the Multilingual BERT Embedding Space (2022.findings-acl)
Copied to clipboard
| Challenge: | Existing studies have explored the advantages of multilingual pre-trained models in capturing shared linguistic knowledge. |
| Approach: | They investigate the anisotropic embedding space and outlier dimensions of the multilingual BERT model for two known issues of the monolingual models. |
| Outcome: | The proposed model has no outlier dimension and has highly anisotropic space . the results show that increasing the isotropy of multilingual space can improve its representation power and performance, similar to what had been observed for monolingual CWRs on semantic similarity tasks. |